Anatomy of the BIRN The Biomedical Informatics Research Network

Transcription

1 Anatomy of the BIRN The Biomedical Informatics Research Network Dr. Philip Papadopoulos University of California San Diego and the San Diego Supercomputer Center BIRN Coordinating Center Co-Investigator NBCR Co-investigator

2 The BIRN Collaboratory Today Enabling collaborative research at 28 research institutions comprised of 37 research groups.

3 What is BIRN A Physical and Geographically-Distributed test bed for a Biomedical Knowledge Infrastructure Creation/Support of Federated Bioscience Databases Data Integration to Enable New Queries among Disparate Data Sources Interoperable Domain Analysis Tools Scalable, Extensible, Replicable Driven by Research needs Pull, not Technology Push A Domain-Specific Grid ~30 Institutions

4 Major Components 3 Distributed Test beds (Scientific Drivers) Mouse, Morph, Function BIRN IT Coordinating Center (@ UCSD) Distributed Hardware Well-defined, Evolving Software Stack on All Machines OS, Middleware, Applications Centralized/BIRN-Wide Services Grid Glue BIRN Portal Information/News/Application Starting Point People-to-people communication, organization Outreach, Training

5 Scientific/Research Goals Drive the BIRN Infrastructure BIRN is Pushing on the How of Cyberinfrastructure NBCR and BIRN are intimately related on a variety of levels sharing increases as you get closer to the IT Infrastructure

6 Why Biomedical Informatics & Research Bio-Complexity Discovery and Systems Research Approaches Complement Hypothesis-based Research Integrative, Multidisciplinary Team Approach Adapted for Complex Queries Versus Focused Approach for Hypothesis-driven Research Team Approach more Dependent on Advanced Technologies and Instrumentation Which Generate Large Data Sets Information Management is a Critical Capability for Biomedical Research in 21st Century and Beyond

7 Data Integrated Across Scales Enable new understanding of the brain by linking data about macroscopic brain function to its molecular and cellular underpinnings Federate Distributed Multiscale Brain Map Data Accommodate associated Large Scale Computational Challenges Provide Infrastructure for Construction of more Accurate Models and more Realistic Simulations of Brain Activity

8 Challenges of Large and Distributed Data Dr. Art Toga (UCLA) was one of the first to articulate the magnitude of the challenge of human brain data - Large-Scale data points and comparisons among 100s to 1000s is needed

9 BIRN Testbeds - Overview Morphometry BIRN Brain Structure in AD, MCI, Depression Function BIRN Activation Differences in Schizophrenia Mouse BIRN High Resolution Imaging and Animal Models of Human Diseases BIRN-CC Coordinating Center for Cyberinfrastructure (Extending to others Groups)

10 MIRIAD Distributed Analysis Deidentified Data from Duke Retrospective Archive Loaded in BIRN Data Grid UCLA LONI Pipeline Register Probabilistic Anatomy Atlas to Subjects Lobar Analysis BWH/MIT 3D Slicer Image Analysis and Segmentation UCSD Clusters Use Teragrid for very large runs Statistical Analysis Detailed Clinical Database Multi-Institutional Research in the Analysis of Depression (MIRIAD) UCLA AIR Registration and Lobar Analysis Duke Archives 1 BWH Probabilistic Atlas (one time transfer) 2 3 Duke Clinical Analysis BWH Intensity Normalization and EM Segmentation UCSD Cluster Computing MIRIAD Data Flow 1) Retrospective data upload from Duke 2) Lobar analysis and Registration of Atlas to Subjects 3) Anatomical Segmentation 4) Comparison to Clinical History 4

11 MIRIAD Project: Accomplishments Improved computational capabilities Segmentation Duke BIRN-MIRIAD Item (semi-automated) (fully-automated) # of tissue classes 3 (Fig1) 23 (Fig2) Time for 200 brains 400 hours 1 hour Time for 200 lobe & 250 hours all lobes (Fig3) and 27 regional analysis regions included above 1 2 3

12 fbirn Multi-Site Data Normalization Reference Anatomical Scan fmri Scans from 10 Different Sites Same Subject, Registered, Same Slice Notice the obvious differences in image across different scanners.

13 Mouse BIRN - Dimensional Scales of Analysis UCLA (LONI) U. Tenn Duke (CIVM) Cal Tech (Beckman) UCSD (NCMIR)

14 Autosegmentation- Morph- Mouse BIRN

15 An Exercise for the Reader There exists a large body of useful middleware It s assembly, hardening and extension into a useful system is left as an exercise to the reader The BIRN is the reader

16 The IT Underwear BIRN Science Teams Challenge Every Facet of Current Grid Technology Large Data Storage/Movement Integration of Data across Sites Network capacity, real transfer rates Computational Resources All-to-All Data Comparisons Fluid Deformations, Automated Tissue Segmentation Security, Integrity, Provenance (History) of Data and Transformations Ease of Use

17 BIRN Evolution: JSR-168 Portlets gridsphere (GSI-aware) Portlet Container User-Defined Layouts and Capability Integrate both BIRN and non-birn capabilities

18 We Began with Standard Hardware Jumpstarted BIRN for functionality Software footprint managed by the BIRN-CC Integration of domain tools, middleware, OS, updates, and more Expansion/upgrade of existing sites have more generic (and less expensive) hardware e.g. Opteron/EM64T, Multi-vendor HW costs $100K/Site four years ago, now $10K-$15K

19 Racks are Replicated at Each Major BIRN Site 20+ Distinct Installations, 100 s of Individual Machines

20 Software Problem in a Nutshell Data & Network CPU Security Enable Analysis of Distributed Biomedical Data in a National-Scale Production Facility Data Sets are Large Data Sets are Many Enable New Queries that Integrate Multiple Sources Specialized Application Codes (from Test Beds) need to work on BIRN-accessible Data Some Analysis Pipelines Require Significant Computation Privacy, Patient Anonymity Required Institutional Ownership of Originals Easily Replicate Entire Software Stack (Including Centralized Services) for other Groups

21 Major System Components Collaborating Groups of Biomedical Researchers Complete Workflows Application Portal Command/Batch Access Domain Application Tools Distributed Data (Collections) Distributed Data (file system) Computation/Analysis Facilities Identity/Login Management Data Integration Mechanisms Authorization and Role Definition Integrated SW Distribution Overall Operations Note: Similar Structure as Many Other Grids

22 Specific Implementations Mouse, Function, Morphometry (+ New Areas and Users ) Pegasus, Kepler, and others BIRN Portal Command/Batch Access E.g. AFNI, Air, 3DSlicer, LONI,.. GSI-Based. GAMA + MyProxy BIRN Data Integration Storage Resource Broker (SRB) AFS (file system) (SSHFS usermount) Condor,Globus: Local clusters + Teragrid SRB for Access Control to Data Semi-Annual BIRN SW Distribution (April/Oct) BIRN-CC Note: Similar Structure as Many Other Grids

23 Global Elements Data location Storage Resource Broker Meta data catalog Data-type aggregation, cross-correlation, integration BIRN Data Mediator Identity Management Use Grid Security Infrastructure (GSI) Public Key System Integrated Grid Accounts Management Architecture (GAMA) from SDSC for ease-of-use and Single Sign On Portal Services Based on GridSphere Dedicated Compute Cluster (32 nodes) + ability to use larger non-dedicated Condor pool

24 More Global Elements Administrative Subset Grid Certificate Authority Condor Collector Integration of computing elements in Runnable/Accessible state for compute jobs Dynamic system nucleated from the BIRN Dedicated Compute Cluster Can integrate (Using Condor-G) with larger infrastructures like Teragrid Network Operations Center (NOC) Styled functions Performance Analysis VPN Services for secure remote administration and data collection Hardware/Network health monitoring BIRN Software Version Control (CVS) Server Rocks Central for Network-based system installation YUM Server (Interim Security Updates) Mail-list services, Video Teleconference Multipoint (MCU)

25 Reality: Substantial Hidden Infrastructure needed to function BIRN Integrated Services (Existence is made as Transparent as Possible)

26 Example: BIRN Login

27 Login Behind the Scenes Login Portlet (standalone functionality that can be rendered by the Portal itself) is active when user gives username/password 1. Secure request is sent to BIRN GAMA Server GAMA Validates user name/password and returns a time-limited GSI proxy credential 2. GridSphere intializes the User Session Loads user preferences for layout of session Holds proxy credential in per-session memory for use to authenticate the user to other grid services 3. Portlets that require user identification can use GSI proxy (single Sign-on)

28 Data is The #1 issue Generic Data issues Key Middleware Issues Authenticated/Authorized Sharing Location and (High-performance) Access when using Distributed Resources Auditing Use. Insuring Data Integrity Replication BIRN-specific Mediation Of Different Data Representations Semantics of the data is key Some Specific meta data: Patient with identity protection MRI Pulse Sequence, Magnet Description

29 Practical Whole System Management The BIRN Software Stack is Large, Complex and Evolutionary Others need to be able to replicate ALL OF BIRN without spending man years of effort The Physical Hardware is Cheap Especially relative to the cost of a person Throwing System Administrators at the problem (the Microsoft/IBM/DBA Model) is not the answer All deployed components of BIRN are Appliances BIRN Uses the Rocks Clustering Toolkit at the Glue to define all System Configurations

30 Replicability Deployed as complete environment Site Racks and Centralized Services Others should be able to replicate all of BIRN Goal is all of BIRN is represented as complete software-only solution BIRN began with Specific HW for sites and has now relaxes that to multi-vendor We use Rocks as a key packaging/deployment vehicle Open Source (Enterprise Linux) Deployed on over 1000 clusters worldwide BIRN needs have substantially affected its design

31 Basic Software Stack for Cluster/Distributed System

32 Common to Every Networked System

33 Rolls Break Apart Software Stacks into Logical and Re-combinable Components Rocks Base HPC Web Server BIRN Base Free- Surfer BIRN Extensions

34 Software Complexity is Reality BIRN-Specific Rolls Rocks-developed Rolls

35 BIRN Software Releases BIRN Rolls Integrates All Software Defines Multiple Appliances Grid POP Compute Servers Application Servers Metadata Catalog Grid Certificate Server Portal Servers Network Analysis Systems Database Servers Goal: A single person should be able to Build all of BIRN The Entire Structure must be Replicable

36 Supporting and Evolving the Deployed Infrastructure MONITORING ASSESSING TECHNICAL OPTIONS FOR BIRN TO BUILD WORKING SYSTEMS SCALABLE SOFTWARE DISTRIBUTION

37 BIRN Now Provides High Performance Connectivity Between Distributed Resources (Computation and Data Storage) JHU Utilizing Teragrid Resources Pulling Data from SRB Secure Access to Large Volumes of Distributed Data Seamless Access to Distributed High Performance Computing Resources Condor is a key abstraction Frameworks (Standards, APIs, Services) and Software for Integration and Interoperation An Evolving Software and Service Structure to meet the needs of Biomedical Researchers

38 Summary BIRN Infrastructure is Driven By the Needs of Key Test beds BIRN-CC is Integrating, Extending, and Evolving an end-to-end Infrastructure Built to enable new interactions Build rapidly, deploy, monitor and test to uncover the real problems The CC is not alone in this endeavor and we actively work with other groups to get to best of breed IT Challenges are formidable (and satisfying)