CyberSKA: Project Update. Cameron Kiddle, CyberSKA Technical Coordinator

Size: px

Start display at page:

Download "CyberSKA: Project Update. Cameron Kiddle, CyberSKA Technical Coordinator"

Tyler Harrison
5 years ago
Views:

1 CyberSKA: Project Update Cameron Kiddle, CyberSKA Technical Coordinator

2 What is CyberSKA? Initiative to develop a scalable and distributed cyberinfrastructure platform to meet evolving science needs of data-intensive radio astronomy on route to the SKA Project participants: University of Calgary, University of British Columbia, University of British Columbia Okanagan, McGill University, Cornell University, National Research Council Canada, IBM Canada, Calgary Scientific Canadian funding for CyberSKA provided by: CANARIE (NEP - Network Enabled Platforms program) Cybera Starting by establishing cyberinfrastructure to support current largescale astrophysical data needs generated by SKA Pathfinder survey projects including: PALFA (Arecibo) GALFACTS (Arecibo) Deep Polarization Field surveys (EVLA, GMRT) Oct 17-21,

3 Experience/Background Leveraging knowledge and experience of the Grid Research Centre at the University of Calgary, IBM, Calgary Scientific and a large technical team (~30 people or ~10 FTE) Started off project by adapting, customizing and extending technologies used by GeoChronos ( a platform developed by the Grid Research Centre enables Earth observation scientists to access and share data and applications and collaborate more effectively employs social networking, cloud computing and data management technologies Making use of existing tools and technologies where possible Elgg, PureWeb, irods, JUpload, CADC Download Manager, Condor, Oct 17-21,

4 Key Documents Documents being submitted as supporting documentation for Software and Computing CoDR Summary Document: WP TR-001 Case Study: CyberSKA A Cyberinfrastructure Platform for Data-Intensive Radio Astronomy Design Documents: Sep. 27, 2010 CyberSKA High Level Architecture v1.1 Apr. 29, 2011 CyberSKA Data Management Architecture v1.0 Oct. 18, 2011 CyberSKA Data Processing Architecture v1.0 Paper: C. Kiddle, A. R. Taylor, J. Cordes, O. Eymere, V. Kaspi, E. Rosolowsky, I. Stairs and A. G. Willis. CyberSKA: An On-line Collaborative Portal for Data- Intensive Radio Astronomy. To appear in Proceedings of the Gateway Computing Environments Workshop, Seattle, Nov. 18, (An SC11 Workshop) Website: Further details on CyberSKA project progress Oct 17-21,

5 High Level Architecture Oct 17-21,

6 Accomplishments Collaborative Portal Distributed Data Management System Data Access Tool Data Query and Processing Service On-line Image Viewer Third Party Application Interface Oct 17-21,

Collaborative Portal Portal built on top of the Elgg open source social networking platform Provides many Facebook-like features including: tags,

7 Collaborative Portal Portal built on top of the Elgg open source social networking platform Provides many Facebook-like features including: tags, bookmarks, profiles, blogs, wikis, contacts, groups, document sharing, discussions, messaging, calendars, status, activity feeds Oct 17-21,

Distributed Data Management System Based on irods (Integrated Rule-Oriented Data System) Abstracts data location from user Supports data replication and

Java Web Start based Supports large data uploads/downloads Improved reliability Multiple file upload Automated mime type recognition For many common file

8 Distributed Data Management System Based on irods (Integrated Rule-Oriented Data System) Abstracts data location from user Supports data replication and cross-site backups Efficient transfer of data using multiple TCP streams Rule engine to automate various tasks Advanced upload/download tools Java Applet / Java Web Start based Supports large data uploads/downloads Improved reliability Multiple file upload Automated mime type recognition For many common file types FITS and Measurement Set (CASA) image data UVFITS and Measurement Set visibility data Automated header extraction and thumbnail generation Oct 17-21,

9 Data Access Tool Access/download subsets of large data sets based on selected parameters and multi-dimensional region of interest Requested data generated in virtualized Condor pool on server side Oct 17-21,

Data Query and Processing Service PostgreSQL database for image metadata (Adherent to VO metadata standards) Query service with RESTful API (spatial, temporal and spectral queries supported) Workflow

10 Data Query and Processing Service PostgreSQL database for image metadata (Adherent to VO metadata standards) Query service with RESTful API (spatial, temporal and spectral queries supported) Workflow service supports mosaicing, plane extraction, image segmentation and spatial reprojection of images returned by query Future workflow modules to include image statistics, Fourier transforms, Oct 17-21,

On-line Image Viewer Client side based image viewer Runs in user s browser Supports interactive panning & zooming, histogram correction, color map adjustments, display pixel data value,

on server, but accessed from user s browser Developed in collaboration with Calgary Scientific, using their PureWeb technology Currently supports interactive panning and zooming,

11 On-line Image Viewer Client side based image viewer Runs in user s browser Supports interactive panning & zooming, histogram correction, color map adjustments, display pixel data value, region statistics, multiple coordinate systems, grids, selection of frame for multi-dimensional images, 2D Gaussian fitting, permalink, screenshots Server side based image viewer Runs on server, but accessed from user s browser Developed in collaboration with Calgary Scientific, using their PureWeb technology Currently supports interactive panning and zooming, histogram correction, color map adjustments, display pixel data value, intensity profiles, movie player, sharing of sessions Multiple platforms supported - Flash, ios, Android Oct 17-21,

Current applications include PALFA Candidate Viewer, PALFA Top Candidates, PALFA Observation

12 Third Party Application Interface API for integrating third party / remotely hosted applications Single sign-on to applications enabled using Oauth Push/pull information and data to/from portal Current applications include PALFA Candidate Viewer, PALFA Top Candidates, PALFA Observation Scheduler, PALFA Diagnostics Tool, GALFACTS Pipeline, Visibility Data Processing Pipeline, Source Counts, Oct 17-21,

13 Portal Usage ~200 members from around the world 30+ groups (GALFACTS, PALFA, EVLA Deep Polarization Field, GMRT Deep Polarization Field, CASA Users, MeqTrees Users, ) Oct 17-21,

14 Next Steps / Sustainability Next Steps Evolve functionality of data management, data processing and visualization tools Continue development of collaboration features based on user feedback Support interoperability with IVOA standards Deploy cloud computing environment and integrate HPC facilities Expand and deploy CyberSKA platform at international sites Support adoption of CyberSKA by industry consortia Establish new industry partners and explore commercialization opportunities Sustainability Current project funding by CANARIE terminates at end of 2011 CANARIE currently waiting for word on its mandate renewal (current mandate ends Mar. 31, 2012) Plan to pursue additional CANARIE funding after their mandate renewal Exploring other funding opportunities also Open to establishing international partnerships with other groups Oct 17-21,

Contact Information Portal: http://www.cyberska.

15 Contact Information Portal: Oct 17-21,

Case Study: CyberSKA - A Collaborative Platform for Data Intensive Radio Astronomy

Case Study: CyberSKA - A Collaborative Platform for Data Intensive Radio Astronomy Outline Motivation / Overview Participants / Industry Partners Documentation Architecture Current Status and Services