Solving informatics challenges to advance plant ecology: a vision for the next 100 years

Transcription

1 Solving informatics challenges to advance plant ecology: a vision for the next 100 years Brian McGill ESA Baltimore August 14, 2015

2 5 10 First point

3 Where I m coming from 1990s work in computer consulting world doing datawarehousing (building databases for business analytics) 2000s analyze BBS and other large datasets 2010s part of numerous ecoinformatics projects BIEN Botanical Information and Ecology Network 3 iterations 20 million records ETE Evolution of Terrestrial Ecosystems E&O Environment and Organisms Consultation with lots of others

4 The trend Papers in ecology with 'database*'

5 Why the rise of ecoinformatics? push The capacity of hard disks double every 23 months Data growing >exponentially Soberón & Peterson (2004)

6 Why the rise of ecoinformatics? pull Conservation & policy Large scales Long time series Inventory mentality Basic ecology Macroecology Ecosystems ecology Global change

7 The data Biotic measurements Abiotic (environmental measurements Time Time Site Site

8 McGill s 9 easy steps of ecoinformatics Collect Scrub Join Store Update Analyze Manage Staff Fund

9 McGill s 9 easy steps of ecoinformatics The real order a 1b Collect Scrub Join Store Update Analyze Manage Staff Fund

10 McGill s 9 easy steps of ecoinformatics Scr Amount of Work Gartner Group 70% of datwarehousing is in data preparation ub Jo Collect in Store Update Analyze Manage Staff Fund

11 McGill s 9 easy steps of ecoinformatics Collect Scrub Join Store Update Analyze Manage Staff Fund

12 Collecting data sources Some are open and on the web Rest is a complex social process Use evolution of sociality 1. Compel/punish (journal, funder requirements) 2. Trust/reputation (small homogeneous group) 3. Joint fate (group selection) Write out an agreement in advance! Most ecoinformatics projects evolve towards openness

13 McGill s 9 easy steps of ecoinformatics Collect Scrub Join Store Update Analyze Manage Staff Fund

14 Gartner Group 70% of datwarehousing is in data preparation

15 Values 100 cm of rainfall yesterday 0 for NA 1.00 vs 10.0 (transcription errors) Instrument errors Data filling? Space Time 4 dimensions Geocoding (Convention center Baltimore N, W) Geoscrubbing 42,100 for North America 100, 42 for North America 0,0 State centers Best tools, but amazing how often 6/14/2015 vs 2015/6/14 Taxonomy Misspellings Synonymy

16 Synonyms and errors Soberon & Peterson 2004

17 Taxonomic Scrubbing in BIEN 2.5M records 600,000 species in New World! 600,000 names 300,000 standardized names after synonymy and misspelling (fuzzy matching) TNRS service Boyle et al 2013

18 Geoscrubbing Synonymy MEX MX Mexico MEXICO México MÃ xico México Mexi ISO alpha 3 ISO alpha 2 official geonames.org (and gadm.org) name capitalization insensitive geonames.org alternate name recognizable misencoding of México translatable HTML character code not matched 439 country names 62 (14%) unrecognizably misspelled 377 recognized 193 recognized countries (49% synonyms) 43% canonical names

19 GeoScrubbing in BIEN ~ 1/3 of records had no lat/lon (or 0/0) >1/2 had + longitude About 1% had other obvious lat/lon errors 15% not in the right country 25% not in the right state/province 67%*85%*75%=41% correct!

20 McGill s 9 easy steps of ecoinformatics Collect Scrub Join Store Update Analyze Manage Staff Fund

21 Joining data Cleaning & synonyms Pinus strobus in USFIA vs Pinus strobus L. in MOBOT Semantic joining USFIA has # stems per 0.04 ha plot MOBOT has a specimen card/occurrence Record connecting The easy part databases do this well

22 How to assemble data Three approaches: Semantic web automated assembly over network Standards agree on database format, individuals contribute Bottom up (standards committees) Top down Only works for relatively uniform data Datawarehousing specific people pull it together and maintain Cost $ Effectiveness

23 Two laws of scrubbing & joining Gartner s law #1 expect it to be 70% of your work McGill s law #2 expect ~50% of the data to be wrong

24 McGill s 9 easy steps of ecoinformatics Collect Scrub Join Store Update Analyze Manage Staff Fund

25 OLTP vs OLAP OLTP=OnLine Transaction Processing The preeminent need in business (since 1960s) Frequent entry of new facts, frequent recall of individual facts, infrequent sweeping analyses Bank account transactions, phone call recording & billing, order entry, accounting OLAP=OnLine Analytical Processing Growing need in business (since 1980s, coined in 1995) Analysis of information stored in OLTP systems Increasing recognition that OLAP systems are not just a query function on OLTP Datawarehousing, multidimensional databases

26 OLAP context OLTP Database OLAP Database Periodic, Incremental Updates

27 Fear 3 rd normal form All undergraduate computer scientists are told to normalize their database schema They tell all ecologists to do this Ecologists always nod their heads knowingly Except experienced database people know normalization is a trade off gradient and it is really good for OLTP and really bad for OLAP

28 OLAP Schema (Star/Snowflake) =Dimensional modelling Time Order Family Diet Species Species Genus Family Authority Body size Functional group Fact Table Abundance Date recorded Plot Latitude Longitude Elevation Landcover Site MAT MAP Biome

29 Katge et al 2011 (MEE) TRY database

30 Multidimensional Databases the conceptual idea? Biotic measurements Abiotic (environmental measurements Time Time Site Site

31 Spatial databases Spatial indexing on lat/lon SQL extensions Distance, area/length, point in polygon PostGIS SELECT binomial, body_size FROM species WHERE ST_Intersects(geom, ST_GeomFromText('POINT(45-100)',4326)); SELECT state.code, species.binomial FROM states JOIN species ON ST_INTERSECTS(states.geom, species.range);

32 McGill s 9 easy steps of ecoinformatics Collect Scrub Join Store Update Analyze Manage Staff Fund

33 Updating the database Key decision Never (one time) Snapshot (fixed points in time) Continuous Amount of work Benefit More work than you think Expect a 50% reduction each time you update

34 McGill s 9 easy steps of ecoinformatics Collect Scrub Join Store Update Analyze Manage Staff Fund

35 The fun part Lamanna et al 2014

36 Serving data to other scientists Pretty mapped screens are nice But I want a subset of the data! Options Downloadable dumps Query & download RESTFUL API R API

37 McGill s 9 easy steps of ecoinformatics Collect Scrub Join Store Update Analyze Manage Staff Fund

38 Over the wall Scientists

39 Iterative prototyping Initial Analysis & Design User Feedback Software Development

40 This is now 1 2 hours/week of your life

41 Software engineering Use cases Design reviews Version control Separate develop and test Automated testing Bug tracking software Steering committees

42 McGill s 9 easy steps of ecoinformatics Collect Scrub Join Store Update Analyze Manage Staff Fund

43 Students or not students Pros of having students do it Valuable career skill Need to train next generation Cons Often years of work without papers or science Often much time spent on training Computer scientists do cutting edge computer science, not boring data scrubbing & databases Consultants are an underconsidered alternative

44 Student training Not everybody is good at it. Not everybody wants to do it. Not everybody should do it. But the ones who do should be rewarded (or at least not penalized). Paraphrasing Stephen Jackson Ecologists computers or techies ecology? Both but make sure the ecology is in there Every ecology student needs training as a consumer of databases (simple SQL)

45 McGill s 9 easy steps of ecoinformatics Collect Scrub Join Store Update Analyze Manage Staff Fund

46 Funding Good luck! More seriously More grants need to build in data management costs (should NSF give free 15% supplements?) NSF needs to figure way to fund maintaining infrastructure

47 Top community priorities 1. Better scrubbing tools Taxonomy, space, time are pretty standard 2. Developing capacity/training 3. Better spatio temporal tools 4. Imagery/raster tools 5. Broader exposure to more database types 6. Don t reinvent the wheel software best practices exist! 7. Keep the ecology in the drivers seat