sdqa.tex

\section{Science Data Quality Assessment}  \label{sec:sdqa}


\subsection{Operations Readiness Requirement}

The project team shall demonstrate that the integrated LSST system can monitor and assess the quality of all data as it is being collected.

\subsection{Objectives}

Science Data Quality Assessment is made up of a comprehensive system of tools to monitor and assess quality of all data as it is being collected including raw and processed data. The suite of tools have been designed to collect, analyze and record required information to assess the data quality and make that information available to a variety of end users; observatory specialist, observatory scientists, downstream processing, the science planning/scheduling process and science users of the data.

The fast cadence of data collection requires highly automated data diagnostic and analysis methods (such as data mining techniques for finding patterns in large datasets, and various machine learning regression techniques). The Science Data Quality Assessment is mostly be automated, however it includes interactive components allowing further investigation and visualization of SDQA status.

Data quality assessment for Rubin must be carried out at a variety of cadences, which have different goals:

\begin{itemize}

	\item Near real-time assessment of whether the data is scientifically useful;
	\item Monitoring telemetry and imaging data to track the state of the integrated observatory, including the telescope, camera, networks and other supporting systems;
	\item Analysis of the prompt processing properties and performance to determine if the alerts stream meets its requirements; and
	\item Analysis of the data release processing properties and performance to determine if the static sky processing meets its requirements.

\end{itemize}

By the time we make a data release the accumulated data quality analysis must be made available as part of the release artifacts.

\subsubsection{Near Real--time Monitoring \& Assessment of the raw data quality}

The quality assessment of the raw image data combines the results from the state of the telescope, the camera (see below) and technical properties of the images.  Each will be analyzed as it is taken to a measure its technical properties both on the at the Summit Facility using the LSSTCam Diagnostic cluster and from properties determined during the prompt processing for alert production.  Performance properties will be based on measurements and characteristics derived from the images themselves and from daily calibration data, these include:

\begin{itemize}

	\item Sensor readnoise, bias and gain variations, bitwise integrity, etc., from the sensor data;
	\item Properties of the measured PSF, based on the three second moments, or equivalently effective FWHM, e1, e2;
	\item Measured sky background level over the full FPA at amplifier level resoution;
	\item Measured source positions and errors relative to a reference catalog ({\it e.g. Gaia}) to monitor FPA stability and pointing accuracy; and
	\item Measured source fluxes and errors relative to a reference catalogue to monitor system throughput, sensitivity and algorithm processing.

\end{itemize}

At a minimum, these metrics enable the Project to determine if the data are within performance parameters to label the visit as "good".   Tooling will be provided by the Construction Project that enable users to monitor trends in these quantities, e.g., as a function of time and where the telescope is pointing and as a function of position in the focal plane.  A reference set of tools will initially be provided by the LOVE interface along with more detailed analysis tooling (as described below).  In some cases, data from the Rubin Auxiliary Telescope (AuxTel) will be used to interpret trends the LSSTCam data.  The quality analysis needed to determine that the AuxTel is taking sufficiently good data will use the same tooling as provided for the main survey data.

\subsubsection{Longer Term Assessment}

long term monitoring, characterization, and optimization of system performance is handled by the System Performance department in Operations.

\subsubsection{Assessing the quality of the processed data}

The information of the processed data relies on the calibration data products and the pipeline properties. In other words, the data assessment at this stage shall include the correction of the systematic errors.

\subsection{SDQA Tools for analysis}

Science Data Quality Assessment will rely on a suite of tools including as the electronic logging, the engineering facility database (EFD), and the Rubin Science Platform (RSP).  There is also a complementary set data visualization tools to facilitate the understanding of the correlation between the data quality and the observatory state.

These tools include:

\begin{itemize}

	\item LOVE - LSST Observing Visualization Environment includes dashboards and other visualizations of the system state;
	\item RubinTV -- front-end for data quality visualization to support nighttime operations
	\item Engineering Facility Database -- engineering data accessible through Rubin Science Platform and pre-defined dashboards;
	\item Consolidated Database -- relational database that holds items such as the exposure log and provids capability to rendezvous system telemetry and science performance metrics;
	%\item SQuaSH -- the Science Quality System Harness (\citeds{sqr-009});
	\item Sasquatch -- service for metrics and telemetry data; collecting, storing, and querying time-series data
	\item Chronograph -- web-based graphical front-end with dashboards for time series data visualization;
	\item Camera Image Viewer -- pixel-level camera image viewing tool with interactive features;
	\item analysis\_tools -- Science Pipelines package for computing science performance metrics and diagnostic plots as part of image reduction pipeline execution
	\item Plot Navigator -- web-based tool for browsing diagnostic plots of science performance;
	\item Times Square -- service for displaying parameterized Jupyter Notebooks as websites;
	\item Rubin Science Platform (RSP) -- used for investigative ad--hoc analysis (\citeds{LSE-319}); the RSP is accessible via the web and includes a Jupyter notebook aspect and other interactive tools for data visualization.

\end{itemize}

\subsection{Criteria for Completeness Description}

The SDQA capabilities will be considered to be successfully complete upon verification of the System Monitoring and Diagnostics and Image Visualization requirements described in the OSS, including demonstration of this toolset using on-sky observations with LSSTCam.
Each verification ticket has been assigned to a CCR to indicate phasing to support the start of on-sky commissioning and readiness for operations.

%At a minimum, SDQA tooling must monitor and record the following terms of the image quality:

%\begin{itemize}

%	\item PSF FWHM;
%	\item PSF shape ellipticity as described by second moments;
%	\item System wavefront measurements for each visit; and
%	\item Throughput measurements over the entire field of view.

%\end{itemize}

%Tooling for SDQA shall demonstrate the ability to display performance on a visit-by-visit basis as well as being able to show the history of performance metric over a user defined span of time.

\subsection{Pre-Operations Interactions}
The pre-operation interaction include training the observing specialists to recognize and respond to warning and errors from subsystems during nighttime operations as well as data quality anomalies.

\subsection{Artifacts for Completion}

\begin{itemize}

	\item Demonstrated functional tool kit as described above;
	\item Derived reports from the Science Verification/Validation survey(s);
	\item SDQA tooling documentation and code repositories

\end{itemize}