The Mission of IDIES

Size: px

Start display at page:

Download "The Mission of IDIES"

Philip Garrett
5 years ago
Views:

1 2018 Alex Szalay

2 The Mission of IDIES Intellectual leadership in the Science of Big Data, together with MINDS Incubator for data intensive discoveries through disruptive assistance, increase JHU agility in Big Data research projects Vision and oversight of high performance and data intensive computing, operate HPC and Big Data facilities, 100G networks (HORnet) Train the next generation in data analytic skills Support unique data resources which give us competitive advantages and visibility

3 Main Components Six JHU schools participating Interdisciplinary faculty appointments 4+ Bloomberg Distinguished Professors, 7+ junior appts First three in the Mathematics of Big Data -> MINDS Endowed data collections ownership of certain unique data sets gives us visibility and a competitive edge (SDSS, Turbulence, Materials) Postdocs and graduate students Engaged in interdisciplinary research New, crosscutting training on the Science of Big Data π-shaped people Small venture/seed funds for game changing research equipment and rapid, disruptive ideas More than $16.5M IDC generated Started down the path towards a more sustainable future

4 New Disciplines Added Carey Business School joined in 2017 Materials Science (HEMI, MEDE, PARADIM) Smart Cities (helping Baltimore city planning) Social Science (several projects with multi-tb data) Internet of Things (wireless sensor networks) Scalable cancer immunotherapy (towards ~TB/day) Large numerical simulations (2PB+ hosted) => SciServer Interactive collaborative data analytics environment Multi PB databases with hundreds of unique datasets Scalable scripting with ipython, Matlab and R Raised almost $30M in federal funds in 5 years

5 Current SciServer Projects domain sub domain project active new Virgo Data Center Millennium FinTechAI Tweet analysis China Migration Chesapeake Bay Biology Electrophysics Spike Sorting Recount2 RDB VC Img Inst Indra Eagle ScaleFree Business School Social Sciences Life Sciences BigWig NAOJ SDSS Manga Cosmological Simulations Streaming Clustering Baltimore City Planning HIPAA CAAPA TSE LSST Astronomy Observing Virtual Universes Computer Science Cancer Imm. Therapy U01 STScI erosita - GSFC erosita - MPE SkyQuery WFIRST Precision Medicine Kennedy Krieger FragData Neutron Scattering Course Material Big Data in Turbulence Materials Science MEDE AS Education AS Gluseen Earth Sciences Ocean Circulation Fluid Dynamics Turbulence DB Rough Surfaces Paradim Astroinforma tics

SDSS Skyserver Prototype in 21st Century data access 2.6B web hits in 12 years 410M external SQL queries 7,000 papers and 450K citations 7,000,000 distinct users vs.

6 SDSS Skyserver Prototype in 21st Century data access 2.6B web hits in 12 years 410M external SQL queries 7,000 papers and 450K citations 7,000,000 distinct users vs. 15,000 astronomers The emergence of the Internet Scientist The world s most used astronomy facility today Collaborative server-side analysis by 10K astronomers SDSS earned the TRUST of the community

7 EB HITS [MILLIONS] Some SkyServer Metrics Total WWW access 2,605,708,373 Total SQL queries 410,309,546 Total CASJobs queries 36,546,986 Total distinct WWW users 6,887,585 Total distinct SQL users 240,011 Total distinct CASJobs users 9,979 ANNUAL WWW ACCESS NEW CASJOBS USERS PER MONTH NEW USERS PER YEAR 900,000 WWW SQL*10 800, , , , , , , ,

Immersive Turbulence the last unsolved problem of classical physics Feynman Understand the nature of turbulence Consecutive snapshots of a large

Shoot test particles (sensors) from your laptop into the simulation, like in the movie Twister 50TB MHD simulation Now: channel flow 100TB, MHD

8 Immersive Turbulence the last unsolved problem of classical physics Feynman Understand the nature of turbulence Consecutive snapshots of a large simulation of turbulence: 30TB Treat it as an experiment, play with the database! Shoot test particles (sensors) from your laptop into the simulation, like in the movie Twister 50TB MHD simulation Now: channel flow 100TB, MHD 256TB 8K 3 simulation just arrived New paradigm for analyzing simulations! 68 Trillion points delivered in 5 years 650TB of simulations accessible R. Burns, C. Meneveau, T. Zaki, G. Eyink, A. Szalay, E. Vishniac

Cosmology Simulations Millennium DB is the poster child/ success story Built by Gerard Lemson (now at JHU) 600 registered users, 17.3M queries, 287B rows http://gavo.mpa-garching.mpg.

9 Cosmology Simulations Millennium DB is the poster child/ success story Built by Gerard Lemson (now at JHU) 600 registered users, 17.3M queries, 287B rows Data size and scalability PB data sizes, trillion particles of dark matter Indra simulations: 512 different 1 Gpc/h box, particles per simulation 35T particles total, 1.1PB Bridget Falck (JHU), Tamás Budavári (JHU), Shaun Cole (Durham), Daniel Crankshaw (JHU), László Dobos (Eötvös), Adrian Jenkins (Durham), Gerard Lemson (MPA), Mark Neyrinck (JHU), Alex Szalay (JHU), and Jie Wang (Beijing)

10 2PB Ocean Laboratory 1km resolution whole Earth model, 1 year run Collaboration between JHU, MIT, Columbia T. Haine, C. Hill, R. Abernathy, R.Gelderloos, G. Lemson, A. Szalay, NSF $1.8M

11 Materials Science Wide and diverse projects at JHU Hopkins Extreme Materials Institute 10 year, >$70million to work with Army Research Lab Understand high-rate response from atoms to meters Complex variety of tools, techniques and data need to be shared across array of disciplines Paradim ($25m NSF Center): Stress-strain curves, high-speed video & x-ray data, 2D and 3D images with atomic to mm resolution, simulations with different physics at different scales Recent NSF Supplement for data infrastructure Collaboration with NanoHub

12 IDIES in Genomics ARIOC: GPU based aligner 50 times faster than anything else SQL Server BCP format option, supports methylation Terabase Search Engine Parallel SQL server warehouse for 265 genomes 240B short reads in the database, 1s search times SnapTron (in progress) Expression levels for 54,000 full RNA sequences Only place to do lateral searches across all samples at a given location along the genome Created C# code for compressed representation of data taken from the BigWig format in DB (1 week-> 1 min) Linking to CBioPortal

Cancer Immunotherapy Trick the immune system to identify cancer cells Complex challenge, lots of tissue data, multicolor staining, image segmentation and measurement Analysis requires spatial

13 Cancer Immunotherapy Trick the immune system to identify cancer cells Complex challenge, lots of tissue data, multicolor staining, image segmentation and measurement Analysis requires spatial statistics, correlation function Strong similarities to astronomy Increase amount of data collected 1000-fold Soon thousands of tissue samples, PBs of data Pattern recognition problem Scalability challenge

14 Image Mosaic

15 The Road Ahead Increase support with Machine Learning, provide translational to science projects Increase our agility, seed funds, hackathons, Involve more of the younger faculty, get fresh ideas Develop a scale-out, cloud strategy Sharpen our focus, build on our unique strengths

Data-Intensive Science Using GPUs. Alex Szalay, JHU

Data-Intensive Science Using GPUs Alex Szalay, JHU Data in HPC Simulations HPC is an instrument in its own right Largest simulations approach petabytes from supernovae to turbulence, biology and brain