• We present the first very long-term simulations of binary neutron star mergers with piecewise polytropic equations of state and in full general relativity. Our simulations reveal that at a time of 30-50 ms after merger, parts of the star become convectively unstable, which triggers the excitation of inertial modes. The excited inertial modes are sustained up to several tens of ms and are detectable by the planned third-generation gravitational-wave detectors at frequencies of a few kHz. Since inertial modes depend on the rotation rate of the star and they are triggered by a convective instability in the post-merger remnant, their detection in gravitational waves will provide a unique opportunity to probe the rotational and thermal state of the merger remnant. In addition, our findings have implications for the long-term evolution and stability of binary neutron star remnants.
  • In this work we analyze the gravitational wave signal from hypermassive neutron stars formed after the merger of binary neutron star systems, focusing on its spectral features. The gravitational wave signals are extracted from numerical relativity simulations of models already considered by De Pietri et al. [Phys. Rev. D 93, 064047 (2016)], Maione et al. [Classical Quantum Gravity 33, 175009 (2016)], and Feo et al. [Classical Quantum Gravity 34, 034001 (2017)], and allow us to study the effect of the total baryonic mass of such systems (from $2.4 M_{\odot}$ to $3 M_{\odot}$), the mass ratio (up to $q = 0.77$), and the neutron star equation of state, both in equal and highly unequal mass binaries. We use the peaks we find in the gravitational spectrum as an independent test of already published hypotheses of their physical origin and empirical relations linking them with the characteristics of the merging neutron stars. In particular, we highlight the effects of the mass ratio, which in the past was often neglected. We also analyze the temporal evolution of the emission frequencies. Finally, we introduce a modern variant of Prony's method to analyze the gravitational wave postmerger emission as a sum of complex exponentials, trying to overcome some drawbacks of both Fourier spectra and least-squares fitting. Overall, the spectral properties of the postmerger signal observed in our simulation are in agreement with those proposed by other groups. More specifically, we find that the analysis of Bauswein and Stergioulas [Phys. Rev. D 91, 124056 (2015)] is particularly effective for binaries with very low masses or with a small mass ratio and that the mechanical toy model of Takami et al. [Phys. Rev. D 91, 064001 (2015)] provides a comprehensive and accurate description of the early stages of the postmerger.
  • Over the last decade block-structured adaptive mesh refinement (SAMR) has found increasing use in large, publicly available codes and frameworks. SAMR frameworks have evolved along different paths. Some have stayed focused on specific domain areas, others have pursued a more general functionality, providing the building blocks for a larger variety of applications. In this survey paper we examine a representative set of SAMR packages and SAMR-based codes that have been in existence for half a decade or more, have a reasonably sized and active user base outside of their home institutions, and are publicly available. The set consists of a mix of SAMR packages and application codes that cover a broad range of scientific domains. We look at their high-level frameworks, and their approach to dealing with the advent of radical changes in hardware architecture. The codes included in this survey are BoxLib, Cactus, Chombo, Enzo, FLASH, and Uintah.
  • We present a study of the merger of six different known galactic systems of binary neutron stars (BNS) of unequal mass with a mass ratio between $0.75$ and $0.99$. Specifically, these systems are J1756-2251, J0737-3039A, J1906+0746, B1534+12, J0453+1559 and B1913+16. We follow the dynamics of the merger from the late stage of the inspiral process up to $\sim$ 20 ms after the system has merged, either to form a hyper-massive neutron star (NS) or a rotating black hole (BH), using a semi-realistic equation of state (EOS), namely the seven-segment piece-wise polytropic SLy with a thermal component. For the most extreme of these systems ($q=0.75$, J0453+1559), we also investigate the effects of different EOSs: APR4, H4, and MS1. Our numerical simulations are performed using only publicly available open source code such as, the Einstein Toolkit code deployed for the dynamical evolution and the LORENE code for the generation of the initial models. We show results on the gravitational wave signals, spectrogram and frequencies of the BNS after the merger and the BH properties in the two cases in which the system collapse within the simulated time.
  • We present results from three-dimensional general relativistic simulations of binary neutron star coalescences and mergers using public codes. We considered equal mass models where the baryon mass of the two Neutron Stars (NS) is $1.4M_{\odot}$, described by four different equations of state (EOS) for the cold nuclear matter (APR4, SLy, H4, and MS1; all parametrized as piecewise polytropes). We started the simulations from four different initial interbinary distances ($40, 44.3, 50$, and $60$ km), including up to the last 16 orbits before merger. That allows to show the effects on the gravitational wave phase evolution, radiated energy and angular momentum due to: the use of different EOSs, the orbital eccentricity present in the initial data and the initial separation (in the simulation) between the two stars. Our results show that eccentricity has a major role in the discrepancy between numerical and analytical waveforms until the very last few orbits, where "tidal" effects and missing high-order post-Newtonian coefficients also play a significant role. We test different methods for extrapolating the gravitational wave signal extracted at finite radii to null infinity. We show that an effective procedure for integrating the Newman-Penrose $\psi_4$ signal to obtain the gravitational wave strain $h$ is to apply a simple high-pass digital filter to $h$ after a time domain integration, where only the two physical motivated integration constants are introduced. That should be preferred to the more common procedures of introducing additional integration constants, integrating in the frequency domain or filtering $\psi_4$ before integration.
  • We present three-dimensional simulations of the dynamics of binary neutron star (BNS) mergers from the late stage of the inspiral process up to $\sim 20$ ms after the system has merged, either to form a hyper-massive neutron star (NS) or a rotating black hole (BH). We investigate five equal-mass models of total gravitational mass $2.207$, $2.373$, $2.537$, $2.697$ and $2.854 M_\odot$, respectively, and four unequal mass models with $M_{\mathrm{ADM}}\simeq 2.53\ M_\odot$ and $q\simeq 0.94$, $0.88$, $0.82$, and $0.77$ (where $q = M^{(1)}/M^{(2)}$ is the mass ratio). We use a semi-realistic equation of state (EOS) namely, the seven-segment piece-wise polytropic SLyPP with a thermal component given by $\Gamma_{th} = 1.8$. We have also compared the resulting dynamics (for one model) using both, the BSSN-NOK and CCZ4 methods for the evolution of the gravitational sector, and also different reconstruction methods for the matter sector, namely PPM, WENO and MP5. Our results show agreement and high resolution, but superiority of BSSN-NOK supplemented by WENO reconstruction at lower resolutions. One of the important characteristics of the present investigation is that, for the first time, this has been done using only publicly available open source software, in particular, the Einstein Toolkit code deployed for the dynamical evolution and the LORENE code for the generation of the initial models. All of the source code and parameters used for the simulations have been made publicly available. This not only makes it possible to re-run and re-analyze our data; it also enables others to directly build upon this work for future research.
  • This report records and discusses the Third Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE3). The report includes a description of the keynote presentation of the workshop, which served as an overview of sustainable scientific software. It also summarizes a set of lightning talks in which speakers highlighted to-the-point lessons and challenges pertaining to sustaining scientific software. The final and main contribution of the report is a summary of the discussions, future steps, and future organization for a set of self-organized working groups on topics including developing pathways to funding scientific software; constructing useful common metrics for crediting software stakeholders; identifying principles for sustainable software engineering design; reaching out to research software organizations around the world; and building communities for software sustainability. For each group, we include a point of contact and a landing page that can be used by those who want to join that group's future activities. The main challenge left by the workshop is to see if the groups will execute these activities that they have scheduled, and how the WSSSPE community can encourage this to happen.
  • The Sign Learning Kink (SiLK) based Quantum Monte Carlo (QMC) method is used to calculate the ab initio ground state energies for multiple geometries of the H$_{2}$O, N$_2$, and F$_2$ molecules. The method is based on Feynman's path integral formulation of quantum mechanics and has two stages. The first stage is called the learning stage and reduces the well-known QMC minus sign problem by optimizing the linear combinations of Slater determinants which are used in the second stage, a conventional QMC simulation. The method is tested using different vector spaces and compared to the results of other quantum chemical methods and to exact diagonalization. Our findings demonstrate that the SiLK method is accurate and reduces or eliminates the minus sign problem.
  • We present results on the effect of the stiffness of the equation of state on the dynamical bar-mode instability in rapidly rotating polytropic models of neutron stars in full General Relativity. We determine the change in the threshold for the emergence of the instability for a range of the adiabatic $\Gamma$ index from 2.0 to 3.0, including two values chosen to mimic more realistic equations of state at high densities.
  • Modern HPC architectures consist of heterogeneous multi-core, many-node systems with deep memory hierarchies. Modern applications employ ever more advanced discretisation methods to study multi-physics problems. Developing such applications that explore cutting-edge physics on cutting-edge HPC systems has become a complex task that requires significant HPC knowledge and experience. Unfortunately, this combined knowledge is currently out of reach for all but a few groups of application developers. Chemora is a framework for solving systems of Partial Differential Equations (PDEs) that targets modern HPC architectures. Chemora is based on Cactus, which sees prominent usage in the computational relativistic astrophysics community. In Chemora, PDEs are expressed either in a high-level \LaTeX-like language or in Mathematica. Discretisation stencils are defined separately from equations, and can include Finite Differences, Discontinuous Galerkin Finite Elements (DGFE), Adaptive Mesh Refinement (AMR), and multi-block systems. We use Chemora in the Einstein Toolkit to implement the Einstein Equations on CPUs and on accelerators, and study astrophysical systems such as black hole binaries, neutron stars, and core-collapse supernovae.
  • We present results about the effect of the use of a stiffer equation of state, namely the ideal-fluid $\Gamma=2.75$ ones, on the dynamical bar-mode instability in rapidly rotating polytropic models of neutron stars in full General Relativity. We determine the change on the critical value of the instability parameter $\beta$ for the emergence of the instability when the adiabatic index $\Gamma$ is changed from 2 to 2.75 in order to mimic the behavior of a realistic equation of state. In particular, we show that the threshold for the onset of the bar-mode instability is reduced by this change in the stiffness and give a precise quantification of the change in value of the critical parameter $\beta_c$. We also extend the analysis to lower values of $\beta$ and show that low-beta shear instabilities are present also in the case of matter described by a simple polytropic equation of state.
  • Challenges related to development, deployment, and maintenance of reusable software for science are becoming a growing concern. Many scientists' research increasingly depends on the quality and availability of software upon which their works are built. To highlight some of these issues and share experiences, the First Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE1) was held in November 2013 in conjunction with the SC13 Conference. The workshop featured keynote presentations and a large number (54) of solicited extended abstracts that were grouped into three themes and presented via panels. A set of collaborative notes of the presentations and discussion was taken during the workshop. Unique perspectives were captured about issues such as comprehensive documentation, development and deployment practices, software licenses and career paths for developers. Attribution systems that account for evidence of software contribution and impact were also discussed. These include mechanisms such as Digital Object Identifiers, publication of "software papers", and the use of online systems, for example source code repositories like GitHub. This paper summarizes the issues and shared experiences that were discussed, including cross-cutting issues and use cases. It joins a nascent literature seeking to understand what drives software work in science, and how it is impacted by the reward systems of science. These incentives can determine the extent to which developers are motivated to build software for the long-term, for the use of others, and whether to work collaboratively or separately. It also explores community building, leadership, and dynamics in relation to successful scientific software.
  • The Cactus Framework is an open-source, modular, portable programming environment for the collaborative development and deployment of scientific applications using high-performance computing. Its roots reach back to 1996 at the National Center for Supercomputer Applications and the Albert Einstein Institute in Germany, where its development jumpstarted. Since then, the Cactus framework has witnessed major changes in hardware infrastructure as well as its own community. This paper describes its endurance through these past changes and, drawing upon lessons from its past, also discusses future
  • We give an introduction to the Einstein Toolkit, a mature, open-source computational infrastructure for numerical relativity based on the Cactus Framework, for the target group of new users. This toolkit is composed of several different modules, is developed by researchers from different institutions throughout the world and is in active continuous development. Documentation for the toolkit and its several modules is often scattered across different locations, a difficulty new users may at times have to struggle with. Scientific papers exist describing the toolkit and its methods in detail, but they might be overwhelming at first. With these lecture notes we hope to provide an initial overview for new users. We cover how to obtain, compile and run the toolkit, and give an overview of some of the tools and modules provided with it.
  • Monte Carlo approaches to radiation transport have several attractive properties such as simplicity of implementation, high accuracy, and good parallel scaling. Moreover, Monte Carlo methods can handle complicated geometries and are relatively easy to extend to multiple spatial dimensions, which makes them potentially interesting in modeling complex multi-dimensional astrophysical phenomena such as core-collapse supernovae. The aim of this paper is to explore Monte Carlo methods for modeling neutrino transport in core-collapse supernovae. We generalize the Implicit Monte Carlo photon transport scheme of Fleck & Cummings and gray discrete-diffusion scheme of Densmore et al. to energy-, time-, and velocity-dependent neutrino transport. Using our 1D spherically-symmetric implementation, we show that, similar to the photon transport case, the implicit scheme enables significantly larger timesteps compared with explicit time discretization, without sacrificing accuracy, while the discrete-diffusion method leads to significant speed-ups at high optical depth. Our results suggest that a combination of spectral, velocity-dependent, Implicit Monte Carlo and discrete-diffusion Monte Carlo methods represents a robust approach for use in neutrino transport calculations in core-collapse supernovae. Our velocity-dependent scheme can easily be adapted to photon transport.
  • Heterogeneous systems are becoming more common on High Performance Computing (HPC) systems. Even using tools like CUDA and OpenCL it is a non-trivial task to obtain optimal performance on the GPU. Approaches to simplifying this task include Merge (a library based framework for heterogeneous multi-core systems), Zippy (a framework for parallel execution of codes on multiple GPUs), BSGP (a new programming language for general purpose computation on the GPU) and CUDA-lite (an enhancement to CUDA that transforms code based on annotations). In addition, efforts are underway to improve compiler tools for automatic parallelization and optimization of affine loop nests for GPUs and for automatic translation of OpenMP parallelized codes to CUDA. In this paper we present an alternative approach: a new computational framework for the development of massively data parallel scientific codes applications suitable for use on such petascale/exascale hybrid systems built upon the highly scalable Cactus framework. As the first non-trivial demonstration of its usefulness, we successfully developed a new 3D CFD code that achieves improved performance.
  • We describe the Einstein Toolkit, a community-driven, freely accessible computational infrastructure intended for use in numerical relativity, relativistic astrophysics, and other applications. The Toolkit, developed by a collaboration involving researchers from multiple institutions around the world, combines a core set of components needed to simulate astrophysical objects such as black holes, compact objects, and collapsing stars, as well as a full suite of analysis tools. The Einstein Toolkit is currently based on the Cactus Framework for high-performance computing and the Carpet adaptive mesh refinement driver. It implements spacetime evolution via the BSSN evolution system and general-relativistic hydrodynamics in a finite-volume discretization. The toolkit is under continuous development and contains many new code components that have been publicly released for the first time and are described in this article. We discuss the motivation behind the release of the toolkit, the philosophy underlying its development, and the goals of the project. A summary of the implemented numerical techniques is included, as are results of numerical test covering a variety of sample astrophysical problems.
  • Component frameworks are complex systems that rely on many layers of abstraction to function properly. One essential requirement is a consistent means of describing each individual component and how it relates to both other components and the whole framework. As component frameworks are designed to be flexible by nature, the description method should be simultaneously powerful, lead to efficient code, and be easy to use, so that new users can quickly adapt their own code to work with the framework. In this paper, we discuss the Cactus Configuration Language (CCL) which is used to describe components ("thorns'') in the Cactus Framework. The CCL provides a description language for the variables, parameters, functions, scheduling and compilation of a component and includes concepts such as interface and implementation which allow thorns providing the same capabilities to be easily interchanged. We include several application examples which illustrate how community toolkits use the CCL and Cactus and identify needed additions to the language.
  • Assembling simulation software along with the associated tools and utilities is a challenging endeavor, particularly when the components are distributed across multiple source code versioning systems. It is problematic for researchers compiling and running the software across many different supercomputers, as well as for novices in a field who are often presented with a bewildering list of software to collect and install. In this paper, we describe a language (CRL) for specifying software components with the details needed to obtain them from source code repositories. The language supports public and private access. We describe a tool called GetComponents which implements CRL and can be used to assemble software. We demonstrate the tool for application scenarios with the Cactus Framework on the NSF TeraGrid resources. The tool itself is distributed with an open source license and freely available from our web page.
  • We present the first simulations in full General Relativity of the head-on collision between a neutron star and a black hole of comparable mass. These simulations are performed through the solution of the Einstein equations combined with an accurate solution of the relativistic hydrodynamics equations via high-resolution shock-capturing techniques. The initial data is obtained by following the York-Lichnerowicz conformal decomposition with the assumption of time symmetry. Unlike other relativistic studies of such systems, no limitation is set for the mass ratio between the black hole and the neutron star, nor on the position of the black hole, whose apparent horizon is entirely contained within the computational domain. The latter extends over ~400M and is covered with six levels of fixed mesh refinement. Concentrating on a prototypical binary system with mass ratio ~6, we find that although a tidal deformation is evident the neutron star is accreted promptly and entirely into the black hole. While the collision is completed before ~300M, the evolution is carried over up to ~1700M, thus providing time for the extraction of the gravitational-wave signal produced and allowing for a first estimate of the radiative efficiency of processes of this type.
  • We present a simple method for applying excision boundary conditions for the relativistic Euler equations. This method depends on the use of Reconstruction-Evolution methods, a standard class of HRSC methods. We test three different reconstruction schemes, namely TVD, PPM and ENO. The method does not require that the coordinate system is adapted to the excision boundary. We demonstrate the effectiveness of our method using tests containing discontinuites, static test-fluid solutions with black holes, and full dynamical collapse of a neutron star to a black hole. A modified PPM scheme is introduced because of problems arisen when matching excision with the original PPM reconstruction scheme.