methodology.tex

\section{Development Methodology and Prioritization Guidance\label{sec:methdology}}

\subsection{Iterative Development Leveraging Existing Technologies }

The services constructed for the LSST Science Platform will be developed following the iterative Agile methodology. While most of LSST software development follows this approach, adopting it is especially advantageous for user-facing services. There, iterative development and nearly continuous stakeholder feedback can provide guidance as to the details of features to be implemented, the continued validity of the approach taken, and the expected focus of intermediate milestones.

The development of the Portal, Notebook, and Web API Aspects will start from significant existing code bases and prior art.
This is a deliberate approach designed to minimize technical risk and leverage end-user familiarity with these interfaces.
The latter also reduces the barrier to user adoption of the products eventually delivered for LSST.

The \textbf{Portal Aspect} is based on the existing, production quality, archive portal interface developed at IRSA/IPAC --- the \emph{Firefly} toolkit.
The primary challenge is integrating the existing Firefly code, and updating the user experience to conform to anticipated user expectations (e.g., supporting all-sky maps and pan/zoom/click-type exploration).
Consistent with the general philosophy, DM should look at achieving the necessary upgrades by re-using existing well-known libraries and tools (e.g., the Plotly library is being adopted to provide 2D and 3D visualizations of tabular data).

The \textbf{Notebook Aspect} environment will be based on the open-source JupyterLab product delivered and maintained by the Jupyter team.
The development of this Aspect of the LSST Science Platform will focus on deployment and integration with the LSST-specific backend services and other Aspects of the platform, rather than developing new or radically different features within the JupyterLab product.

Finally, the \textbf{Web API Aspect} is envisioned as implementing existing, widely-adopted, community protocols (in particular, ones from the Virtual Observatory suite of protocols and standards).
Similarly to the other Aspects, it will benefit from leveraging existing codes and libraries wherever appropriate.
We view providing a robust TAP (including ObsTAP) service as particularly critical, both for external and internal use.
Other IVOA standards that we expect to support include SCS for simple catalog searches, SIAv2 for image searches, SODA for image cutouts and mosaics, and VOSpace (in addition to WebDAV) for access to user files.
We are exploring the applications of the recently settled DataLink protocol for providing a richer view of LSST's data holdings and their relationships.
We will also ensure that LSST data are well-represented in the IVOA Registries, particularly to enable the convenient discovery of LSST data through non-LSST client tools.

\subsection{Prioritization Guidance}

Here we give some overall feature prioritization guidance, to enable the construction of initial (mostly functional) requirements and intermediate development milestones.

Portal Aspect:
\begin{enumerate}
	\item Deployment of the initial Firefly back-end within the (prototype) LSST Data Access Center at NCSA.
	\item Integration of the initial Firefly front- and back-ends with LSST Science Platform backend services. For example, this includes the authentication and authorization mechanisms, relational databases, file stores, etc.
	\item User experience improvements, such as addition of all-sky maps with pan/zoom/select navigation metaphors, modernization of the look-and-feel, streamlining of the UI and deprecation of rarely used widgets. \textbf{Once this level of functionality is met (at scale), the Portal Aspect will have achieved the minimum level of viability for deployment to operations}.
	\item Improved user workflow integration with other Aspects of the LSST Science Platform. For example, it should be possible to begin data exploration in the Portal Aspect (e.g., by interactively selecting data sets) and seamlessly transfer the sub-selected catalogs and images to the Notebook Aspect's JupyterLab environment for further, more complex, analysis using provided Python libraries.
	\item Addition of new widgets and abilities to the Portal, that address most requested and broadly useful end-user needs.
	\item Widget-level integration with JupyterLab.
\end{enumerate}

Notebook Aspect:
\begin{enumerate}
	\item Deployment of the initial JupyterLab product within the (prototype) LSST Data Access Center at NCSA.
	\item Integration of the JupyterLab product with LSP backend services, most notably authentication and authorization, user management, databases, and file stores. \textbf{Once this level of functionality is met (at scale), the Notebook Aspect will have achieved the minimum level of viability for deployment to commissioning and operations}.
	\item Development of libraries and utilities to ease the submission of user-written code from Jupyter notebooks to the batch compute system.
	\item Creation and curation of a library of 3rd party code that will be made available to LSP end-users.
\end{enumerate}

Web API Aspect:
\begin{enumerate}
	\item Development and deployment of initial data access APIs needed to satisfy the back-end needs of the Portal and Notebook Aspects.
These may not yet ``speak'' the final, standards-compliant, protocols.
	\item Integration of the Web API Aspect with LSP backend services, most notably authentication and authorization, user management, databases, and file stores.
	\item Deployment of key tabular data in IVOA-compliant data models --- particularly image metadata in the ObsCore format.
	\item Deployment of critical protocols (including SCS, TAP, ObsTAP, SIAv2, SODA, VOEvent streaming support, and VO Registry support) at commonly-encountered levels of standards compliance (e.g., the most commonly used ADQL features).
\textbf{Once this level of functionality is met (at scale), the Web API Aspect will have achieved the minimum level of viability for deployment to operations}
	\item Further improvement of standards-compliance as well as deployment of additional standards-compliant protocols, e.g., DataLink, throughout the Web API Aspect, and integration with all other elements of the Platform.
	\item Further conformance of tabular data to IVOA and community data models, notably CAOM2.
\end{enumerate}

It is assumed that the development of backend services will be driven by the needs of the front-end Aspects.