Harmonizing Multi-Model at the World Bank Group

Transcription

1 Harmonizing Multi-Model at the World Bank Group Valentin Prudius, World Bank Group Dagobert Soergel, University at Buffalo & World Bank Group Denisa Popescu, World Bank Group Tyler Replogle, MarkLogic COPYRIGHT 13 June 2017MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

2 A Flexible Database for Harmonizing Multi-Model Data Valentin Prudius, World Bank Group Dagobert Soergel, University at Buffalo & World Bank Group Denisa Popescu, World Bank Group Tyler Replogle, MarkLogic Strictly Confidential 2013

3 Outline Introduction World Bank Group Enterprise Search WB Ontology Implementation in MarkLogic What We ve Learned

4 INTRODUCTION WORLD BANK GROUP

5

6 The World Bank Group GOVERNMENTS PRIVATE SECTOR IBRD IDA IFC MIGA ICSID Middle Income Countries Low Income Countries Foreign and Local Investors Tribunal

7 WBG Today 16,000 nationalities Staff 15,000 LT/ST consultants 150 Country offices 170 Nationalities

8 Supporting 240 locations outside of the U.S.

9 Development Reaching All Sectors rural development technology trade opportunities private sector laws justice youth social development economic growth investments governance partners education community infrastructure health nutrition urban development taxes gender environment institutions sanitation water transparency peace jobs money business voice inclusion security data energy climate

10 History Snapshot 60 s 50 s Infrastructure 90 s 70 s Rural 00 s Social Governance 80 s Health, Education 10 s Data Fiscal

11 Sustainable Development Goals

12 WBGIT: We need to take advantage of emerging technologies Mobile Social Cloud Data

13 ENTERPRISE SEARCH

14 Enterprise Search: Some User Pain Points in WB Is what I find accurate? Whose stuff is this? What are these confusing titles/file names? When was the last time a subject matter expert verified that this was accurate? Why do I find the same document several times? Can't find a document that I know exists. Where is it? Why didn t it show up in the results? Why isn t it in the first 10 pages of results? Where do I look?

15 Current State: WB Information Landscape

16 Challenges Greatest challenge: Content & Relevancy Complex data source ingestion Inconsistent and poorly documented content descriptors Duplicated content within and across repositories Full text data quality Difficulties defining the relevancy algorithm due to unified search for documents, people, and projects

17 Complex Requirements

18 Complex Requirements Title of Presentation

19 Approach Search as an Information Consolidator & Organizer Integrated single results set Results & relevancy in the context of the entire collection Compensate for Data Deficiency Deduplication & Data Harmonization Add Vocabularies & Semantics User activity & content analysis à Relevancy improvement Revised Focus Document Search Administrative & procedural queries WB staff personal queries Documents Data Data Harmonization & Semantics People User Experience Relevancy Interface Consolidated and Organized view Information & User Context Wide spectrum of User Needs

20 WB Ontology Strictly Confidential 2013

21 WBG: An Information Processing Organization Any task will apply all useful internal and external information WB Ontology Data harmonization Open up data silos Data integration Data Find Connect Apply Reference data Unique identifiers Better solutions / decisions

22 WB Ontology A logically coherent entity-relationship model at a general, conceptual level to define all types of data produced or used in the Bank, no matter in what system the data are kept or what data definition language is used. Specification of (1) the things and processes (concrete or abstract) that constitute or are the object of the business of the Bank and (2) the relationships between them Canonical data model for the enterprise not E-R diagrams (limited expressiveness) not restrictive models in specific modeler software

23 Why a World Bank Ontology? needed everywhere Data harmonization and integration for an Integrated enterprise search database Data ingestion from many sources with mapping to the harmonized data model The backbone for a comprehensive index to all data produced or used in the WB Support for text analytics: Entity recognition and relationship extraction Towards a WB knowledge graph Logic for query processing and inference The underlying logic for user interfaces that guide users through tasks, including search, help users navigate in a sea of data A framework in which to place the many WB taxonomies and other controlled lists of entity values A framework for specifying data policies and governance

24 Legal entity Place Topic Person Organization T a x o n o m ie s Gupta Monet Klein Gupta McNeill Chung Org1 WBG Org2 World Bank Org3 IBRD Org4 IDA Org5 IFC Org6 MIGA Org7 ICSID Org8 IMF Org9 UN family Org10 Adeso Europe. Eastern Europe.. Moldova.. Ukraine.. Azerbaijan WB admin regions. Europe & Central Asia.. Moldova.. Russian Federation.. Azerbaijan ICT. ICT applications.. sector-specific... city info syst.... e-commerce... financial MIS... social prot. MIS... transportation IS... e-government Reference data Taxonomies, controlled lists Entity values Unique identifiers

25 WB Ontology Relationship types Person <worksinorganization> (Organization, InOrganizationStatus, InOrganizationRole, JobTitle, Date, Date) Organization <LocatedIn> (Location, LocationRole, LocationLevel, Date, Date) D a t a [McNeill] <worksinorganization> (Org1 [WBG], OfficialUnit, Leader, President, 1968, 1981) [Chung] <worksinorganization> (Org1 [WBG], OfficialUnit, Leader, President, 2012, 9999) Org10 [Adeso] <LocatedIn> (KE, Headquarters, Country, 2012, 9999) Org10 [Adeso] <LocatedIn> (London, Office, City,????, 9999) Databases Statements

26 Two-Level Architecture for the WB Ontology Broad relationships Document <createdby> (LegalEntity, CreatorRole) CreatorRole refines the relationship, it is a Refiner Refinement through relationship argument CreatorRole (selected values) Author, Translator, Illustrator, Editor ContractManager LegalClearer OriginatingUnit Drafter Modifier Filer BlogContact

27 Implementation in MarkLogic Strictly Confidential 2013

28 Ontology to Data Fields / XML Tag Mappings Document <hasassociatedlegalentity> (TTL, Person: TaskManager, TimeSpan) Document <hastitle> (Text, TitleStatus, Language)

29 Data Harmonization Source A Staff Authors Stored as User Identifier delimited by comma EI:490635, Source B Publishing Contact Stored as User Identifier EI: Source C Author or Sender Stored as Last Name, First Name delimited by semicolon EI: Replogle, Tyler; Popescu, Denisa (some records as the delimiter)

30 MarkLogic Features Tagging documents to World Bank Vocabularies Match Making with reverse queries Recency Weight Range index sloping factor Weighting Query classification Reverse query for rule-based classification Proximity Searching Near Queries and distance weighting

31 What We ve Learned Effective enterprise search is more than a technology problem, it is primarily a content problem Leverage MarkLogic to harmonize content in a single model Tune the search and adapt continuously based on user feedback & content characteristics: Relevancy IE: Give more weighting to phrases in the query Text IE: Classify queries as a type of queries IE: Use Several World Bank Vocabularies Promote Content Best bets, direct answers, and baseball cards Upgrade/downgrade content Content Remediation Consumer search experience leads to high expectations that need to be managed Build Learn Measure Release 2 Release 1 Beta Release Release 3