The Large Synoptic Survey Telescope (LSST) is a large-aperture, wide-field,
ground-based survey system that will image the sky in six optical bands from
320 to 1050 nm, uniformly covering approximately $18,000$deg$^2$ of the sky
over 800 times. The LSST is currently under construction on Cerro Pach\'on in
Chile, and expected to enter operations in 2022. Once operational, the LSST
will explore a wide range of astrophysical questions, from discovering "killer"
asteroids to examining the nature of Dark Energy.
The LSST will generate on average 15 TB of data per night, and will require a
comprehensive Data Management system to reduce the raw data to scientifically
useful catalogs and images with minimum human intervention. These reductions
will result in a real-time alert stream, and eleven data releases over the
10-year duration of LSST operations. To enable this processing, the LSST
project is developing a new, general-purpose, high-performance, scalable, well
documented, open source data processing software stack for O/IR surveys.
Prototypes of this stack are already capable of processing data from existing
cameras (e.g., SDSS, DECam, MegaCam), and form the basis of the Hyper-Suprime
Cam (HSC) Survey data reduction pipeline.
The Dark Energy Survey (DES) is a 5000 deg2 grizY survey reaching
characteristic photometric depths of 24th magnitude (10 sigma) and enabling
accurate photometry and morphology of objects ten times fainter than in SDSS.
Preparations for DES have included building a dedicated 3 deg2 CCD camera
(DECam), upgrading the existing CTIO Blanco 4m telescope and developing a new
high performance computing (HPC) enabled data management system (DESDM).
The DESDM system will be used for processing, calibrating and serving the DES
data. The total data volumes are high (~2PB), and so considerable effort has
gone into designing an automated processing and quality control system. Special
purpose image detrending and photometric calibration codes have been developed
to meet the data quality requirements, while survey astrometric calibration,
coaddition and cataloging rely on new extensions of the AstrOmatic codes which
now include tools for PSF modeling, PSF homogenization, PSF corrected model
fitting cataloging and joint model fitting across multiple input images.
The DESDM system has been deployed on dedicated development clusters and HPC
systems in the US and Germany. An extensive program of testing with small rapid
turn-around and larger campaign simulated datasets has been carried out. The
system has also been tested on large real datasets, including Blanco Cosmology
Survey data from the Mosaic2 camera. In Fall 2012 the DESDM system will be used
for DECam commissioning, and, thereafter, the system will go into full science