Particle physics has an ambitious and broad experimental programme for the
coming decades. This programme requires large investments in detector hardware,
either to build new facilities and experiments, or to upgrade existing ones.
Similarly, it requires commensurate investment in the R&D of software to
acquire, manage, process, and analyse the shear amounts of data to be recorded.
In planning for the HL-LHC in particular, it is critical that all of the
collaborating stakeholders agree on the software goals and priorities, and that
the efforts complement each other. In this spirit, this white paper describes
the R&D activities required to prepare for this software upgrade.
The 3.2 giga-pixel LSST camera will produce approximately half a petabyte of
archive images every month. These data need to be reduced in under a minute to
produce real-time transient alerts, and then added to the cumulative catalog
for further analysis. The catalog is expected to grow about three hundred
terabytes per year. The data volume, the real-time transient alerting
requirements of the LSST, and its spatio-temporal aspects require innovative
techniques to build an efficient data access system at reasonable cost. As
currently envisioned, the system will rely on a database for catalogs and
metadata. Several database systems are being evaluated to understand how they
perform at these data rates, data volumes, and access patterns. This paper
describes the LSST requirements, the challenges they impose, the data access
philosophy, results to date from evaluating available database technologies
against LSST requirements, and the proposed database architecture to meet the
The BaBar database has pioneered the use of a commercial ODBMS within the HEP
community. The unique object-oriented architecture of Objectivity/DB has made
it possible to manage over 700 terabytes of production data generated since
May'99, making the BaBar database the world's largest known database. The
ongoing development includes new features, addressing the ever-increasing
luminosity of the detector as well as other changing physics requirements.
Significant efforts are focused on reducing space requirements and operational
costs. The paper discusses our experience with developing a large scale
database system, emphasizing universal aspects which may be applied to any
large scale system, independently of underlying technology used.