VIRTUAL OBSERVATORY TECHNOLOGIES

Transcription

1 VIRTUAL OBSERVATORY TECHNOLOGIES / The Johns Hopkins University

2 Moore s Law, Big Data! 2

3 Outline 3 SQL for Big Data Computing where the bytes are Database and GPU integration CUDA from SQL Data intensive Web services Behind the scenes Working examples Sloan Digital Sky Survey Virtual Observatory tools and services

4 The Virtual Observatory 4 The Virtual Observatory is a framework that enables new astronomical research by greatly enhancing access to worldwide data and computing resources. How it works How to build it How to use it What s next

5 Hierarchy of Services 5 Atomic services Access to observations, simulations Access to models Higher level services Combine for more functionality User and analysis tools Can be a high level service, too

6 Heterogeneous Datasets 6 Blobs: images, spectra, etc... Access, transfer Catalogs Fast searches, indexes

7 Structured Query Language 7 SQL`92 standard Almost in English SELECT <columns> FROM <table> WHERE <conditions> Astronomical Data Query Language An extended subset GIS-like spatial

8 Structured Query Language 8 SQL`92 standard Almost in English SELECT RA, Dec FROM Stars WHERE r < 15 Astronomical Data Query Language An extended subset GIS-like spatial

9 Joining Tables 9 Sources in observations fields: 2 tables SELECT f.fieldid, s.objid, s.ra, s.dec, FROM Fields AS f INNER JOIN Sources AS s ON s.fieldid=f.fieldid WHERE f.exptime > 1000 AND s.rmag > 16

10 Calculations in SQL 10 Computed columns Use J-H in SELECT and/or WHERE Similarly functions, e.g., POWER(10,-0.4*Rmag) Grouping SELECT FieldID, AVG(J), STDEV(J) FROM Sources GROUP BY FieldID Can use for histograming, etc E.g., SDSS Catalog Archive here

11 Surveys in Astronomy 11 Sloan Digital Sky Survey TB Catalog Archive Server Custom tools and indices Upcoming Surveys PanSTARRS: 100TB LSST: 1PB+ 201?

12 New Moore s Law 12 In the number of cores Faster than ever (for now)

13 New Programming Paradigm s of cores 27k parallel threads per GPU Running a billion threads a second Forget the fancy old algorithms Built on wrong assumptions Today CPU is free, RAM is slow GPU has >50GB/s bandwidth Still difficult to occupy the cores

14 Hybrid Architecture 14 launch run sync

15 Extending SQL Server 15 Dedicated service for direct access Shared memory IPC w/ on-the-fly data transform IPC SQL

16 Extending SQL Server 16 Dedicated service for direct access Shared memory IPC w/ on-the-fly data transform IPC SQL

17 Spatial Statistics 17 Correlation functions From pair-counts 8 bins State of the art Dual-tree traversal High resolution bins? Just like brute force

18 Sloan DR bins

19 All Done Inside the Database 19 Pair counts computed on GPU Returns 2D histogram as a table (i, j, cts) Calculate the correlation fn in SQL Can also do async parallel GPU jobs

20 All Done Inside the Database 20 Pair counts computed on GPU Returns 2D histogram as a table (i, j, cts) Calculate the correlation fn in SQL Can also do async parallel GPU jobs

21 21 Distributed Data

22 Data at the Projects 22 Exponential growth Projects last 3-5 years, data sent upwards at the end Data will never be centralized Most data at projects More responsibility on projects Bring analysis close to the data

23 23

24 Data Federation 24 Metcalfe s Law Utility of computer networks grows as the number of possible connections: O(N 2 ) The Virtual Observatory The federation of N astronomy archives has utility O(N 2 ), i.e. possibilities for making discoveries The whole is more than the sum of the parts

25 Interoperability Challenges 25 Metadata standards Data discovery Data requests Data delivery Units Database queries Distributed applications Authentication and authorization

26 US National Virtual Observatory 26 NVO Research NSF ITR Program: $10M for 5 years 17 organizations: Astro, CS, IT VAO Facility NSF $20M for 5 years Operational phase!

27

28

29 IVOA Specifications 29

30 First Standards 30 VOTable Universal container for tables (in XML) First VO standard (from the DTD era) ConeSearch Simple catalog access based on location First VO standard interface (http get) Many implemented them!

31 Early Standards 31 Simple Image Access Protocol (SIAP) Http request, similar to opening a web page Returns links to the matching images in votable Assumes we know how to deal with FITS images Universal Content Descriptor (UCD) Crystallized set of keywords from literature For data discovery not queries

32 Components 32 Discovery Distributed Computing Directory, Sky coverage Web & Grid services Access Tables, Catalogs VOStat Messaging Images, Spectra SAMP, VOPipe Events Distributed Storage User Interfaces Aladin VOSpace Topcat Authentication Mirage, etc

33 33 VO Examples VO Applications and Services

34 NVO Quick Start 34

35 Ready, Steady 35

36 DataScope 36 Collect info in VO On a particular object Or a part of the sky GRBs, transients, etc. VO plotting tools FITS images Catalog data And more

37 Bandpass Services 37 Public repository Search by keyword or eff Extract in various formats Register & submit yours Web site On-the-fly plotting Easy access to all Web services To code against

38 Spectrum Services 38 Public repository SDSS, 2dF spectra, etc Spatial and SQL search Register & submit yours Web site On-the-fly plotting Building composites De-reddening Line analysis Web services

39 Open SkyQuery 39 SkyNode interface to archives Implements ADQL returns VOTable Basic node understands REGION Full node understands XMATCH SkyQuery portal Knows the SkyNodes from Registry Understands federated query

40 WESIX 40 Web Enabled Source-Identification with Crossmatching Higher level astronomy services built on other existing VO services: SExtractor service and Open SkyQuery Result can be sent to plotting tool for quick inspection.

41 VOStat 41 Enabling R For VO data

42 Sky Coverage 42 Discovery

43 Transients: VOEvent 43

44 Help! 44

45 45 VO for Developers Automated tools for analysis Advanced services

46 Web Services 46 Simple HTTP requests ConeSearch Simple Image Access Standard SOAP and REST Interoperable across platforms IVOA compliant XML messages Programming toolkits exist

47 Command Line: VO-CLI 47 VOTool

48 Command Line: VO-CLI 48 VOTool

49 49 Future New features Better integration

50 VOSpace Storage instances soon everywhere Save intermediate data products Arrange for their transfer to other places VOPipe Chain VOSpaces for data flow between services Async execution of custom processing steps

51 Summary 51 More and Moore data: new opportunities No central data store but at projects On-site processing: CPU + GPU Hierarchical Services Standardized interfaces Data federation New VxOs VaO: Virtual Astronomical Observatory VsO,

52 Sites to Explore 52

53 53