Science User Interface and Tools: Status. David R. Ciardi & Xiuqin Wu On Behalf of the SUIT Team at IPAC

Transcription

1 Science User Interface and Tools: Status David R. Ciardi & Xiuqin Wu On Behalf of the SUIT Team at IPAC 1

2 Building the SUIT Vision Given the speed with which web technologies evolve, SUIT was intentionally left undesigned at FDR time. Start of construction marked the beginning of fully defining the scope and design of the SUIT Developed a more complete understanding of the DM system context for the SUIT Developed a vision for the SUIT to drive the design 2

3 Purpose of the SUIT Enable access to the LSST data products Enable visualization and exploration of the LSST data Provide an interface to Level 3 facilities enabling addedvalue processing and analysis close to the data" 3

4 Philosophy of the SUIT The SUIT needs to be simple enough to engage the novice and general users while flexible enough to meet the needs of the experienced users DM will build a environment that fulfills the needs of the general user (e.g., portal and workspace, searching, image visualization, table manipulation, plotting, etc.) Components usable by others to build tools that meet their own special needs Enable creativity and flexibility SUIT is a portal, a set of tools, a workspace, and a toolkit of JS and Python APIs all designed to mate the user with the data. 4

5 User and Community Resources Authentication/Authorization Layer User Oriented Conceptual Diagram Data LSST Entry Web Portal Visualization Server (Firefly) LSST Data Access Layer: User Specific Code Workspace Resources: storage, processing, local db Images, Catalogs, L3, Calibration etc. 5

6 Turning Concept into Requirements Analyzed each of the pre-fdr 180+ SUIT requirements in the context of the newly developed vision for SUIT ~85% of the requirements were updated to be more clear and consistent with each other and the vision (15% of old requirements did not support vision) Added ~20% new requirements to reflect the SUIT vision for interactive analysis and usability from Python, unifying the web portal, user workspace, and Level 3 processing environments Delivered a draft to SUIT working group to be reviewed by DM Team 6

7 Turning Concept and Requirements into a Design Working to drive concept and requirements into an architecture that is implementable Working to ensure SUIT is integrated in the overall DM environment Drive a strong connection between visualization tools and Python 7

8 What is Firefly? Image and Data visualization suite Client Server architecture Data are maintained on server Enables fast display on client side Unified data model (brushing & linking: images, tables, plots, histograms) Modular components (images, tables, plots, histograms) Operational in Infrared Science Archive at IPAC 8

9 Firefly modularity enables custom environments 9

10 Visualization & Python We ve made the first steps in the integration of the Firefly visualization components with Python Enables complex processing and analysis beyond the visualization tools: Creativity and Flexibility Callable from a Python command-line prompt or from within a Jupyter Notebook Calculational capabilities of the SUIT extensible by providing for standard ways to invoke Python code in response to GUI actions. Currently in the prototype stage to inform the requirements, the design and architecture, and the development plan 10

11 Python Command & Firefly 11

12 Jupyter Notebook Widget 12

13 Firefly & Python Data Sharing 13

14 SUIT Support for Project Use Working with other groups within in DM and the Project Internal project needs DM Pipeline: support data visualization in pipeline development SDQA: support data analysis and quality assessment Camera group: support image visualization Telescope and Site: support image visualization Learn from people working with the data what they need to make the SUIT responsive to the needs of the users 14

15 Image Mask Viewer Firefly Python API LSST image: mask plane taken from data Detected (red) Crosstalk (green) Edge (blue) Intrp (yellow) User control Color Transparency Visibility 15

16 Camera Group: Image Visualization 16

17 Guiding Development with A Prototype Data Access Center Enable an end-to-end integration of (initially) the Data Access, SUIT, and Middleware/Infrastructure components of DM into a cooperating system. Enable operational and scaling tests of the above. Expose the Data Access (e.g., database query) and SUIT components to a very limited science user community for early feedback on the design and implementation. This is an engineering facility and not intended to be a full production-level data archive service Services may go up and down Code and functionality will change 17

18 PDAC Deployments 2016/2017: Version : Version 3 SDSS Stripe 82 data Web portal and API data access 2017: Version 2 WISE time domain data (100 Billion rows) Workspace environment Versions 2, 3, 4 are tentative scenarios, not defined plans; more details will become available pending conclusion of DM replan Inclusion of even larger public datasets (e.g., maybe PanSTARRS) LSST processed data Improved and increased SUIT functionality 2019: Version 4 Support for Commission and analysis of commissioning data Transitioning over to operational DAC for 2020/2021 support of science verification and operations 18

19 PDAC Access Policy (LDM-482) Project team members who specifically request it and receive approval from DM management, after consultation with the teams building and operating the components. Astronomers with LSST data rights not employed by the project ("external users"), who submit an application substantively describing investigations they wish to perform, including a description of how they plan to provide technical and usability feedback to the project, and expressing their understanding that the PDAC is not a production system. The PDAC will initially only be able to accommodate a small number of external users 19

20 FY17: SUIT Construction Finish the major portion of SUIT system design, including integration with user workspace Deploy PDAC web portal v1: provide data search/display/visualization of SDSS stripe82 data v2: add initial workspace capabilities and WISE/NEOWISE time domain data Deliver science application developer support tools started in FY2016 Firefly Python API and Jupyter widgets Further develop visualization capabilities for Camera I&T needs 20

21 FY18: SUIT Construction Commissioning Support: Deploy web portal and API to support searches and display of the data generated by ComCam Visualize the full raft images, drill down to CCD, and amplifier level Support interactive, user-driven commissioning data analyses Connect LSST data to reference catalogs (e.g., Gaia) Update PDAC with new capabilities and data; solicit user feedback 21

22 FY19: SUIT Construction Full camera commissioning support Visualize the full focal plane images, drill down to raft, CCD, and/or amplifier level (multi-resolution images display) Advanced integration between SUIT portal, API, and workspace Advanced light curve visualization Deploy the beta version of alert subscription/notification system Update PDAC with new capabilities and data; solicit user feedback 22

23 FY20: SUIT Construction Science Verification support Level-3 data support Scientific analysis tools Scalability of deployment of SUIT Manual, Help Desk, knowledge base, documentation, tutorials, Full DAC commissioning; continue to obtain user feedback 23

24 FY21/22: SUIT Construction Science Operations support Provide API for alert system subscription and access to alert history Remaining advanced visualization and analysis tools Refine and finalize the DAC capabilities Community support for use of SUIT and development of custom environments Address any remaining issues and capabilities Address issues found in testing Community support for use of SUIT and development of custom environments 24

25 Summary SUIT effort will deliver a web-based user interface allowing the community to visualize, explore, process, and analyze the LSST data, as well as advanced visualization components enabling close-to-the-data analysis in Level 3. Strong Python and visualization connection Integrated workspace and portal environments Python and JS APIs for user customization Incremental development and integration and testing through PDAC 25